Region-of-interest based video encoding

ABSTRACT

A video processing unit can include a non-object-based region-of-interest detection neural network, a threshold selection module and a region-of-interest map generator. The non-object-based region-of-interest detection neural network can be configured to receive a video frame and generate a plurality of candidate non-object-based region-of-interest blocks. The threshold selection module can be configured to receive the plurality of candidate non-object-based region-of-interest blocks and identify a plurality of selected region-of-interest blocks based on a predetermined threshold. The region-of-interest map generator can be configured to receive the selected non-object-based region-of-interest blocks and generate a region-of-interest map.

BACKGROUND OF THE INVENTION

Numerous techniques are used for reducing the amount of data consumed bythe transmission or storage of video. One common technique is to usevariable bit rate encoding of video frame data. For example, a firstbitrate can be utilized to encode one or more regions of interest (ROI),and a second bitrate can be utilized to encode one or more non-regionsof interest.

Referring to FIG. 1, a video processing system, according to theconventional art, is shown. The video processing system can include anartificial intelligence (AI) accelerator 110, a central processing unit120 and a video encoder 130. In one implementation, the artificialintelligence (AI) accelerator 110 can be a graphics processing unit(GPU). The artificial intelligence (AI) accelerator 110 can include anobject-based region-of-interest (OB-ROI) detector neural network 140.The object-based region-of-interest (OB-ROI) detector neural network 140can receive a stream of video frames 150. The object-basedregion-of-interest (OB-ROI) detector neural network (NN) 140 can beconfigured to generate a plurality of candidate object-basedregion-of-interest (OB-ROI) blocks 160. In one implementation, theobject-based region-of-interest (OB-ROI) detector neural network 140 canbe a deep neural network (DNN) including a regression network fordetermining object-based region-of-interest (NOB-ROI) blocks 160 and aclassification network for object detection. In one implementation, thecandidate object-based region-of-interest (OB-ROI) blocks 160 caninclude a determined probability of the regions of interest (e.g.,confidence score), an object type for the regions of interest, and thelike. In one implementation, the associated probability can comprise theprobability that the given region of interest comprises at least aportion of an object of a given object type. For example, theobject-based region-of-interest (OB-ROI) detector neural network 140 candetermine if the memory block includes an object of one of a pluralityof object types, and a probability the of object type. Accordingly, theterm object-based region-of-interest (OB-RIO) detection as used hereingenerally refers identification of objects within a data set that alsoincludes identification of an associated object type. Memory blocks thatdo not include an object of one of the plurality of object types can beclassified as a non-region of interest. The memory block may be a 16×16,64×64, or the like matrix of pixel values. Object classification canconsume a substantial amount of processing bandwidth on the artificialintelligence (AI) accelerator. For example, classification ofmobilenet_v2 based stream of video frames on a typical graphicsprocessing unit (GPU) can consume approximately 12% of the processingbandwidth of the graphics processing unit (GPU), and a correspondingamount of power consumption.

The central processing unit 120 can include a sorting and non-maximumsuppression (NMS) module 170. The sorting and non-maximum suppression(NMS) module 170 can receive the plurality of candidate object-basedregion-of-interest (OB-ROI) blocks 160. The plurality of candidateobject-based region-of-interest (OB-ROI) blocks 160 can comprise asubstantial amount of data that can consume a substantial amount ofcommunication bandwidth between the artificial intelligence (AI)accelerator 110 and the central processing unit (CPU) 120, and orconsume a substantial amount of power to transmit the data between theartificial intelligence (AI) accelerator 110 and the central processingunit (CPU) 120. The sorting and non-maximum suppression (NMS) module 170can be configured to sort the candidate object-based region-of-interest(OB-ROI) blocks 160 for each object type based on the associatedprobability. For example, the plurality of candidate object-basedregion-of-interest (OB-ROI) blocks 160 can include hundreds, thousandsor more candidates that are sorted by the corresponding confidence scoreof the candidate object-based region-of-interest (OB-ROI) blocks 160.The sorting and non-maximum suppression (NMS) module 170 can also beconfigured to combine multiple overlapping object-basedregion-of-interest (OB-ROI) blocks 160 to determine one or moreregion-of-interest bounding boxes. For example, the candidate memoryblock (MB) for a given object type with the highest confidence score canbe selected as the initial decided bounding box. Each candidate memoryblock (MB) of the same object type with a next lower confidence score iscompared to the current decided bounding box to determine how much theyoverlap. If the current candidate memory block (MB) overlaps with thecurrent decided bounding box by more than a predetermined amount (e.g.,50%), the current candidate memory block (MB) is disregarded. If thecurrent candidate memory block (MB) overlaps with the current decidedbounding box by less than the predetermined amount, the currentcandidate memory block (MB) is added to the current decided boundingbox. The candidate memory blocks (MB) are processed until one boundingbox is determined for each object type. Accordingly, the termnon-maximal suppression as used herein generally refers to a functionthat iteratively performs an intersection over union of a plurality ofcandidate blocks to determine a region-of-interest. The sorting andnon-maximum suppression (NMS) can also consume a substantial amount ofprocessing bandwidth on the central processing unit 120. For example,sorting and non-maximum suppression (NMS) of the mobilenet_v2 basedstream of video frames on a typical central processing unit such as aXeon8163 processor can consume approximately 10% of the processingbandwidth of the central processing unit (CPU).

The video encoder 130 can be configured to generate a compressed bitstream 180 based on the determined one or more region-of-interestbounding boxes. In one implementation, the video encoder 130 can beconfigured to encode the data in the one or more region-of-interestbounding boxes at a first bit rate and one or morenon-regions-of-interest at a second bit rate, wherein the first bit rateis greater than the second bit rate. In another implementation, thevideo encoder 130 can be configured to encode the data in the one ormore region-of-interest bounding boxes at a first quality and the one ormore non-regions-of-interest at a second quality.

The object-based region-of-interest detection for use with variable rateencoding can be computationally intensive. Accordingly, there is acontinuing need for improved variable bit rate encoding of video images.

SUMMARY OF THE INVENTION

The present technology may best be understood by referring to thefollowing description and accompanying drawings that are used toillustrate embodiments of the present technology directed towardregion-of-interest based video encoding techniques.

In one embodiment, a video processing unit can include an artificialintelligence accelerator including a non-object-based region-of-interestdetection neural network, a threshold selection module, and aregion-of-interest map generator. The non-object-basedregion-of-interest detection neural network can be configured to receivea video frame and generate a plurality of candidate non-object-basedregion-of-interest blocks. The threshold selection module can beconfigured to receive the plurality of candidate non-object-basedregion-of-interest blocks and identify a plurality of selectedregion-of-interest blocks based on a predetermined threshold. Theregion-of-interest map generator can be configured to receive theselected non-object-based region-of-interest blocks and generate aregion-of-interest map. The artificial intelligence accelerator canoptionally also include an object-based region-of-interest detectionneural network configured to receive the video frame and generate aplurality of candidate object-based region-of-interest blocks. Theartificial intelligence accelerator is configured to selectivelygenerate the plurality of candidate non-object-based region-of-interestblocks by the non-object-based region-of-interest detection neuralnetwork or generate the plurality of candidate object-basedregion-of-interest blocks by the object-based region-of-interestdetection neural network. When candidate object-based region-of-interestblocks are generated by the object-based region-of-interest detectionneural network, a sorting and non-maximum suppression (NMS) module on acentral processing unit can be configured to sort the plurality ofcandidate object-based region-of-interest blocks for respective objecttypes based on associated probabilities, and combine multipleoverlapping object-based region-of-interest blocks to determine one ormore region-of-interest bounding boxes for respective object types. Avideo encoder can differentially encode the video frame to generate acompressed bit stream based on the region-of-interest map or the one ormore region-of-interest bounding boxes.

In another embodiment, a method of video processing can includegenerating a plurality of candidate non-object-based region-of-interestblocks for a video frame. The candidate non-object-basedregion-of-interest blocks can each include a corresponding confidencescore. A plurality of selected region-of-interest blocks comprisingcandidate non-object-based region-of-interest blocks each havingconfidence scores greater than a predetermined threshold score can beselected. A region-of-interest map can be generated based on theplurality of selected region-of-interest blocks.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present technology are illustrated by way of exampleand not by way of limitation, in the figures of the accompanyingdrawings and in which like reference numerals refer to similar elementsand in which:

FIG. 1 shows a block diagram of a video processing system, according tothe conventional art.

FIG. 2 shows a block diagram of a video processing system, in accordancewith aspects of the present technology.

FIG. 3 shows a block diagram of a region-of-interest (ROI) map generatormodule, in accordance with aspects of the present technology.

FIGS. 4A-4D illustrate an exemplary workflow within an artificialintelligence (AI) accelerator, in accordance with aspects of the presenttechnology.

FIG. 5 shows a block diagram of a video processing system, in accordancewith aspects of the present technology.

FIG. 6 shows a block diagram of a video processing system, in accordancewith aspects of the present technology.

FIG. 7 shows a block diagram of an exemplary processing unit including avideo processing unit, in accordance with aspects of the presenttechnology.

FIG. 8 shows a block diagram of an exemplary processing core, inaccordance with aspects of the present technology.

DETAILED DESCRIPTION OF THE INVENTION

Reference will now be made in detail to the embodiments of the presenttechnology, examples of which are illustrated in the accompanyingdrawings. While the present technology will be described in conjunctionwith these embodiments, it will be understood that they are not intendedto limit the technology to these embodiments. On the contrary, theinvention is intended to cover alternatives, modifications andequivalents, which may be included within the scope of the invention asdefined by the appended claims. Furthermore, in the following detaileddescription of the present technology numerous specific details are setforth in order to provide a thorough understanding of the presenttechnology. However, it is understood that the present technology may bepracticed without these specific details. In other instances, well-knownmethods, procedures, components, and circuits have not been described indetail as not to unnecessarily obscure aspects of the presenttechnology.

Some embodiments of the present technology which follow are presented interms of routines, modules, logic blocks, and other symbolicrepresentations of operations on data within one or more electronicdevices. The descriptions and representations are the means used bythose skilled in the art to most effectively convey the substance oftheir work to others skilled in the art. A routine, module, logic blockand/or the like, is herein, and generally, conceived to be aself-consistent sequence of processes or instructions leading to adesired result. The processes are those including physical manipulationsof physical quantities. Usually, though not necessarily, these physicalmanipulations take the form of electric or magnetic signals capable ofbeing stored, transferred, compared and otherwise manipulated in anelectronic device. For reasons of convenience, and with reference tocommon usage, these signals are referred to as data, bits, values,elements, symbols, characters, terms, numbers, strings, and/or the likewith reference to embodiments of the present technology.

It should be borne in mind, however, that these terms are to beinterpreted as referencing physical manipulations and quantities and aremerely convenient labels and are to be interpreted further in view ofterms commonly used in the art. Unless specifically stated otherwise asapparent from the following discussion, it is understood that throughdiscussions of the present technology, discussions utilizing the termssuch as “receiving,” and/or the like, refer to the actions and processesof an electronic device such as an electronic computing device thatmanipulates and transforms data. The data is represented as physical(e.g., electronic) quantities within the electronic device's logiccircuits, registers, memories and/or the like, and is transformed intoother data similarly represented as physical quantities within theelectronic device.

In this application, the use of the disjunctive is intended to includethe conjunctive. The use of definite or indefinite articles is notintended to indicate cardinality. In particular, a reference to “the”object or “a” object is intended to denote also one of a possibleplurality of such objects. The use of the terms “comprises,”“comprising,” “includes,” “including” and the like specify the presenceof stated elements, but do not preclude the presence or addition of oneor more other elements and or groups thereof. It is also to beunderstood that although the terms first, second, etc. may be usedherein to describe various elements, such elements should not be limitedby these terms. These terms are used herein to distinguish one elementfrom another. For example, a first element could he termed a secondelement, and similarly a second element could be termed a first element,without departing from the scope of embodiments. It is also to beunderstood that when an element is referred to as being “coupled” toanother element, it may be directly or indirectly connected to the otherelement, or intervening element may be present. In contrast, when anelement is referred to as being “directly connected” to another element,there are not intervening elements present. It is also to be understoodthat the term “and or” includes any and all combinations of one or moreof the associated elements. It is also to be understood that thephraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting.

Referring to FIG. 2, a video processing system, in accordance withaspects of the present technology, is shown. Operation of the videoprocessing system will be further explained with reference to FIGS. 4Athrough 4D, which illustrates an exemplary workflow within theartificial intelligence (AI) accelerator. The video processing system200 can include an artificial intelligence (AI) accelerator 210communicatively coupled to a video encoder 215. In one implementation,the artificial intelligence (AI) accelerator 210 can be a graphicsprocessing unit (GPU), a neural processing unit (NPU), a vectorprocessor, a memory processing unit, or the like, or combinationsthereof. The artificial intelligence (AI) accelerator 210 can include anon-object-based region-of-interest (NOB-ROI) detection neural network(NN) 220, a threshold selection module 225 and a region-of-interest(ROI) map generator 230. The non-object-based region-of-interest(NOB-ROI) detection neural network (NN) 220 can receive a stream ofvideo frames 215. For example, a video frame as illustrated in FIG. 4Acan be received. The non-object-based region-of-interest (NOB-ROI)detection neural network (NN) 220 can be configured to generate aplurality of candidate non-object-based region-of-interest (NOB-ROI)blocks 240. For example, the non-object-based region-of-interest(NOB-ROI) detection neural network (NN) can generate a plurality ofcandidate non-object-based region-of-interest (NOB-ROI) blocks 410 asillustrated in FIG. 4B. In one implementation, the non-object-basedregion-of-interest (NOB-ROI) detection neural network (NN) 220 can be adeep neural network (DNN) including a regression network for determiningthe non-object-based region-of-interest (NOB-ROI) blocks 240 andcorresponding probability of interest (e.g., confidence score) for eachnon-object-based region-of-interest (NOB-ROI) blocks 240. For example,the object-based region-of-interest (OB-ROI) detector neural network 140can determine a probability that the respective memory blocks (MB) areinteresting.

The threshold selection module 225 can receive the plurality ofcandidate non-object-based region-of-interest (NOB-ROI) blocks 240. Thethreshold selection module 225 can be configured to identify a pluralityof selected region-of-interest (ROI) blocks 245 each of which have aprobability greater than a predetermined threshold. For example, memoryblocks having a probability greater than a predetermined threshold canbe identified as non-object-based region-of-interest (NOB-ROI) blocks420, as illustrated in FIG. 4C. In one implementation, thenon-object-based region-of-interest (NOB-ROI) blocks 420 can correspondportion of the image including a person. The memory blocks having aprobability less than a predetermined threshold can be identified asnon-regions of interest 430. The non-regions of interest 423 cancorrespond to background portions of the image. In one implementation,the threshold selection module 225 can indicate selected memory block(MB) that have a probability of the regions of interest greater than 1%,or other specified threshold probability. For example, if a given memoryblock (MB) has a probability greater than the threshold, the givenmemory block (MB) can be indicated as a non-object-basedregion-of-interest (NOB-ROI). However, an associated object type is notdetermined for the non-object-based region-of-interest (NOB-ROI).Accordingly, the term non-object-based region-of-interest (NOB-RIO)detection as used herein generally refers identification of objectswithin a data set without the identification of an associated objecttype. If the given memory block (MB) has a probability less than thethreshold, the given memory block (MB) can be indicated as a non-regionof interest. In one implementation regions-of-interest can correspond tothe foreground, and the non-regions-of-interest can correspond to thebackground in the video frame.

In other implementations, the threshold selection module 225 can beconfigured to identify region-of-interest (ROI) blocks 245 having aprobability in a plurality of a predetermined threshold ranges. Forexample, memory blocks (MB) having an associated probability of greaterthan 20% can be identified as a first level of region-of-interest,memory blocks (MB) having an associated probability between 1% and 20%can be identified as a second level of region-of-interest, and memoryblocks (MB) having an associated probability less than 1% can beidentified as non-regions-of-interest.

In contrast, to the conventional video processing system, the candidatenon-object-based region-of-interest (NOB-ROI) blocks 240 are nottransmitted to a central processing unit (CPU). In addition, sorting andnon-maximum suppression (NMS) of the candidate non-object-basedregion-of-interest (NOB-ROI) blocks 240 is not performed. Therefore, thevideo processing system in accordance with aspects of the presenttechnology can reduce communication bandwidth utilization and or reducepower consumption associated with the data transfer. Likewise, the videoprocessing system in accordance with embodiment of the presenttechnology can reduce central processor unit (CPU) utilization and orpower consumption associated with processing by the central processorunit (CPU).

The region-of-interest (ROI) map generator module 230 can receive theplurality of selected region-of-interest (ROI) blocks 245. Theregion-of-interest (ROI) map generator module 230 can be configured togenerate a region-of-interest (ROI) map 250 including an indicator forregion-of-interest (ROI) blocks that are of interest or not. Theregion-of-interest (ROI) map can include identifiers of data blocks of adata set and identifiers of selected and non-selected non-object-basedregion-of-interest (NOB-ROI). For example, the region-of-interest (ROI)map can include an identifier 440 of each memory block in the image, andan identifier 450 indicated if the memory block is a selected blockrepresenting a non-object-based region-of-interest (NOB-ROI) 420 or anon-selected block representing a non-region-of-interest 430, asillustrated in FIG. 4D.

Referring now to FIG. 3, a region-of-interest (ROI) map generator module230, in accordance with aspects of the present technology, is shown. Theregion-of-interest (ROI) map generator module 230 can loop over theblocks of the video frames 310 to determine if given memory blocks areidentified as selected non-object-based region-of-interest blocks. Forexample, the region-of-interest (ROI) map generator module 230 candetermine if the given memory bock is indicated to be a selected block.If the given memory block is not a selected memory block, acorresponding region-of-interest (ROI) bit in the region-of-interest mapcan be set to a first value 330. If the given memory block is a selectedmemory block, the corresponding region-of-interest (ROI) bit in theregion-of-interest map can be set to a second value 340. For example, ifthe given memory block (MB) is not a selected memory block, aregion-of-interest (ROI) bit in the memory map corresponding to thegiven memory block (MB) can be set to ‘0.’ If the given memory block(MB) is a selected memory block, a region-of-interest (ROI) bit in thememory map corresponding to the given memory block (MB) can be set to‘1.’ For multi-level region-of-interest indications, a correspondingmulti-bit value in the region-of-interest (ROI) map can be set to acorresponding bit value.

Referring again to FIG. 2, the video encoder 215 can receive the streamof video frames 235 and the region-of-interest (ROI) map 250. The videoencoder 215 can be configured to generate a compressed bit stream 255based on the region-of-interest (ROI) map 250. In one implementation,the video encoder 215 can be configured to encode the memory blockshaving a corresponding first region-of-interest (ROI) map bit value(e.g., non-region-of-interest memory blocks) at a first bit rate (e.g.,low bit rate) and the memory blocks having a corresponding secondregion-of-interest (ROI) map bit value (e.g., region-of-interest memoryblocks) at a second bit rate (e.g., high bit rate). For example, thenon-object-based region-of-interest (NOB-ROI) blocks 410 illustrated inFIG. 4B can be encoded at a high bit rate, while the non-region ofinterest blocks 420 can be encoded at a lower bit rate. In anotherimplementation, the video encoder 215 can be configured to encode thememory blocks having a corresponding first region-of-interest (ROI) mapbit value (e.g., non-region-of-interest memory blocks) at a firstquality (e.g., low quality) and the memory blocks, having acorresponding second region-of-interest (ROI) map bit value (e.g.,region-of-interest memory blocks) at a second quality (e.g., highquality).

The non-object-based region-of-interest (NOB-ROI) video processingsystem, in accordance with aspects of the present technology,advantageously reduce the computational workload because the objectclassification of conventional object-based region-of-interest (OB-ROI)does not need to be performed for variable rate video encoding. Inaddition, the reduced computational workload can result in a reductionin power consumption by the video processing system 200. Thenon-object-based region-of-interest (NOB-ROI) can also advantageously beperformed entirely in the artificial intelligence (AI) accelerator 210.In non-object-based region-of-interest (NOB-ROI), there is no need forsorting and non-maximum suppression (NMS) and therefore thecomputational workload of the central processing unit 120 can bereduced. The reduction of the computational workload of the centralprocessing unit 120 can also reduce power consumption in the centralprocessing unit 120. In addition, the bandwidth utilization of one ormore communication links between the artificial intelligence (AI)accelerator 210 and the central processing unit 120 is advantageouslyreduced because data does not need to be transferred from the artificialintelligence (AI) accelerator 210 to the central processing unit 120 forperforming sorting and non-maximum suppression (NMS). The reduction indata transmission from the artificial intelligence (AI) accelerator 210to the central processing unit 120 can also reduce power consumption.

Referring now to FIG. 5, a video processing system, in accordance withaspects of the present technology. The video processing system 500 caninclude an artificial intelligence (AI) accelerator 505, a centralprocessing unit 510 and a video encoder 515. The artificial intelligence(AI) accelerator 505 and the central processing unit 510 can beselectively configurable to detect non-object-based regions-of-interest(NOB-ROI) or object-based regions-of-interest (OB-ROI) in a stream ofvideo frames.

For non-object-based regions-of-interest (NOB-ROI) detection, theartificial intelligence (AI) accelerator 505 can include anon-object-based region-of-interest (NOB-ROI) detection neural network(NN) 520, a threshold selection module 525 and a region-of-interest(ROI) map generator module 530. The non-object-based region-of-interest(NOB-ROI) detection neural network (NN) 520 can receive a stream ofvideo frames 535. The non-object-based region-of-interest (NOB-ROI)detection neural network 520 can be configured to generate a pluralityof candidate non-object-based region-of-interest (NOB-ROI) blocks 540.In one implementation, non-object-based region-of-interest (NOB-ROI)detection neural network 520 can be a deep neural network (DNN)including a regression network for determining the non-object-basedregion-of-interest (NOB-ROI) blocks 540 and corresponding probability ofinterest (e.g., confidence score) for each non-object-basedregion-of-interest (NOB-ROI) block 540. For example, thenon-object-based region-of-interest (NOB-ROI) detector neural network520 can determine a probability that the respective memory blocks (MB)are interesting.

The threshold selection module 525 can receive the plurality ofcandidate non-object-based region-of-interest (NOB-ROI) blocks 540. Thethreshold selection module 525 can be configured to identify a pluralityof selected region-of-interest (ROI) blocks 545 each of which have aprobability greater than a predetermined threshold. In oneimplementation, the threshold selection module 525 can indicate selectedmemory block (MB) that have a probability of the regions of interestgreater than 1%, or other specified threshold probability. For example,if a given memory block (MB) has a probability greater than thethreshold, the given memory block (MB) can be indicated as aregion-of-interest. If the given memory block (MB) has a probabilityless than the threshold, the given memory block (MB) can be indicated asa non-region of interest. In other implementations, the thresholdselection module 525 can be configured to identify selectedregion-of-interest (ROI) blocks 545 having probabilities in a pluralityof a predetermined threshold ranges. For example, memory blocks (MB)having an associated probability of greater than 20% can be identifiedas a first level of region-of-interest, memory blocks (MB) having anassociated probability between 1% and 20% can be identified as a secondlevel of region-of-interest, and memory blocks (MB) having an associatedprobability less than 1% can be identified as non-regions-of-interest.

In contrast, to the conventional video processing system, the candidatenon-object-based region-of-interest (NOB-ROI) blocks 540 are nottransmitted to a central processing unit (CPU). In addition, sorting andnon-maximum suppression (NMS) of the candidate non-object-basedregion-of-interest (NOB-ROI) blocks 540 is not performed. Therefore, thevideo processing system in accordance with aspects of the presenttechnology can reduce communication bandwidth utilization and or reducepower consumption associated with the data transfer. Likewise, the videoprocessing system in accordance with embodiment of the presenttechnology can reduce central processor unit (CPU) utilization and orpower consumption associated with processing by the central processorunit (CPU).

The region-of-interest (ROI) map generator 530 can receive the pluralityof selected region-of-interest (ROI) blocks 545. The region-of-interest(ROI) map generator 530 can be configured to generate aregion-of-interest (ROI) map 550 including an indication forregion-of-interest (ROI) blocks that are of interest or not. Theregion-of-interest (ROI) map generator 530 can generate theregion-of-interest (ROI) map 550 as described above with reference toFIG. 3.

The video encoder 515 can receive the stream of video frames 535 and theregion-of-interest (ROI) map 550. The video encoder 515 can beconfigured to generate a compressed bit stream 555 based on theregion-of-interest (ROI) map 550. In one implementation, the videoencoder 515 can be configured to encode the memory blocks having acorresponding first region-of-interest (ROI) map bit value (e.g.,non-region-of-interest memory blocks) at a first bit rate (e.g., low bitrate) and the memory blocks having a corresponding secondregion-of-interest (ROI) map bit value (e.g., region-of-interest memoryblocks) at a second bit rate (e.g., high bit rate). In anotherimplementation, the video encoder 515 can be configured to encode thememory blocks having a corresponding first region-of-interest (ROI) mapbit value (e.g., non-region-of-interest memory blocks) at a firstquality (e.g., low quality) and the memory blocks having a correspondingsecond region-of-interest (ROI) map bit value (e.g. region-of-interestmemory blocks) at a second quality (e.g., high quality).

For non-object-based regions-of-interest (NOB-ROI) detection, theartificial intelligence (AI) accelerator 505 can further include anobject-based region-of-interest detector neural network (NN) 560. Theobject-based region-of-interest detector neural network 560 can receivethe stream of video frames 535. The object-based region-of-interest(OB-ROI) detector neural network (NN) 560 can be configured to generatea plurality of candidate object-based region-of-interest (OB-ROI) blocks565. In one implementation, the object-based region-of-interest (OB-ROI)detector neural network 560 can be a deep neural network (DNN) includinga regression network for determining object-based region-of-interest(NOB-ROI) blocks 565 and a classification network for object detection.In one implementation, the candidate object-based region-of-interest(OB-ROI) blocks 565 can include a determined probability of the regionsof interest (e.g., confidence score), an object type for the regions ofinterest, and the like. In one implementation, the associatedprobability can comprise the probability that the given region ofinterest comprises at least a portion of an object of a given objecttype. For example, the object-based region-of-interest (OB-ROI) detectorneural network 560 can determine if the memory block includes an objectof one of a plurality of object types, and a probability of the objecttype. Memory blocks that do not include an object of one of theplurality of object types can be classified as a non-regions ofinterest. Each memory block can be a predetermined matrix size (e.g.,16×16, 64×64, etc.) of pixel values.

The central processing unit 510 can include a sorting and non-maximumsuppression (NMS) module 570. The sorting and non-maximum suppression(NMS) module 570 can receive the plurality of candidate object-basedregion-of-interest (OB-ROI) blocks 565. The sorting and non-maximumsuppression (NMS) module 570 can be configured to sort the candidateobject-based region-of-interest (OB-ROI) blocks 565 of each object typebased on the associated probabilities. For example, the plurality ofcandidate object-based region-of-interest (OB-ROI) blocks 565 caninclude hundreds, thousands or more candidates that are sorted by thecorresponding confidence score of the candidate object-basedregion-of-interest (OB-ROI) blocks 565 for each of the different objecttypes. The sorting and non-maximum suppression (NMS) module 570 can alsobe configured to combine multiple overlapping object-basedregion-of-interest (OB-ROI) blocks 565 to determine one or moreregion-of-interest bounding boxes. For example, the candidate memoryblock (MB) with the highest confidence score can be selected as theinitial decided bounding box for a given object type. Each candidatememory block (MB) of the same object type with a next lower confidencescore is compared to the current decided bounding box to determine howmuch they overlap. If the current candidate memory block (MB) overlapswith the current decided bounding box by more than a predetermine amount(e.g., 50%), the current candidate memory block (MB) can be disregarded.If the current candidate memory block (MB) overlaps with the currentdecided hounding box by less than the predetermined amount, the currentcandidate memory block (MB) can be added to the current decided boundingbox. The candidate memory blocks (MB) are processed until one boundingbox is determined for each object type.

For non-object-based regions-of-interest (NOB-ROI) detection, the videoencoder 515 can be configured to generate the compressed bit stream 555based on the determined one or more region-of-interest bounding boxes.In one implementation, the video encoder 515 can be configured to encodethe data in the one or more region-of-interest bounding boxes at a firstbit rate and one or more non-regions-of-interest at a second bit rate,wherein the first bit rate is greater than the second bit rate. Inanother implementation, the video encoder 515 can be configured toencode the data in the one or more region-of-interest bounding boxes ata first quality and the one or more non-regions-of-interest at a secondquality.

For non-object-based region-of-interest (NOB-ROI) detection, thecomputational workload of the video processing system 500 can be reducedbecause the object classification does not need to be performed forvariable rate video encoding. In addition, the reduced computationalworkload can result in a reduction in power consumption by the videoprocessing system 500. The non-object-based region-of-interest (NOB-ROI)can also advantageously be performed entirely in the artificialintelligence (AI) accelerator 505. In non-object-basedregion-of-interest (NOB-ROI), there is no need for sorting andnon-maximum suppression (NMS) and therefore the computational workloadof the central processing unit 510 can be reduced. The reduction of thecomputational workload of the central processing unit 120 can alsoreduce power consumption in the central processing unit 510. Inaddition, the bandwidth utilization of one or more communication linksbetween the artificial intelligence (AI) accelerator 505 and the centralprocessing unit 510 is advantageously reduced because data does not needto be transferred from the artificial intelligence (AI) accelerator 505to the central processing unit 510 for performing sorting andnon-maximum suppression (NMS). The reduction in data transmission fromthe artificial intelligence (AI) accelerator 505 to the centralprocessing unit 510 can also reduce power consumption.

Referring now to FIG. 6, a video processing system, in accordance withaspects of the present technology. The video processing system 600 caninclude an artificial intelligence (AI) accelerator 605, a centralprocessing unit 610 and a video encoder 615. The artificial intelligence(AI) accelerator 605 and the central processing unit 610 can beselectively configurable to detect non-object-based regions-of-interest(NOB-ROI) or object-based regions-of-interest (OB-ROI) in a stream ofvideo frames.

For non-object-based regions-of-interest (NOB-ROI) detection, theartificial intelligence (AI) accelerator 605 can include anon-object-based region-of-interest (NOB-ROI) detection neural network620, a threshold selection module 625 and a region-of-interest (ROI) mapgenerator module 630. The non-object-based region-of-interest (NOB-ROI)detection neural network (NN) 620 Call receive a stream of video frames635. The non-object-based region-of-interest (NOB-ROI) detection neuralnetwork (NN) 620 can be configured to generate a plurality of candidatenon-object-based region-of-interest (NOB-ROI) blocks 640. In oneimplementation, non-object-based region-of-interest (NOB-ROI) detectionneural network 620 can be a deep neural network (DNN) including aregression network for determining the non-object-basedregion-of-interest (NOB-ROI) blocks 640 and corresponding probability ofinterest (e.g., confidence score) for each non-object-basedregion-of-interest (NOB-ROI) block 640. For example, thenon-object-based region-of-interest (NOB-ROI) detector neural network620 can determine a probability that the given memory block (MB) isinteresting.

The central processing unit 610 can include a simple sorting module 675.The simple sorting module 675 can receive the plurality of candidatenon-object-based region-of-interest (OB-ROI) blocks 640. The simplesorting module 675 can be configured to sort the candidatenon-object-based region-of-interest (OB-ROI) blocks 640 based on theassociated probabilities. The candidate non-object-basedregion-of-interest (OB-ROI) blocks sorted based on the associatedprobabilities can be received by the threshold selection module 625. Thethreshold selection module 625 can be configured to identify apredetermined number of the plurality of selected region-of-interest(ROI) blocks 645 having the highest associated probability. For example,the 100 memory blocks (MB) with die highest associated probability canbe indicated as regions-of-interest. The other memory blocks (MB) can beindicated as a non-regions of interest.

The region-of-interest (ROI) map generator module 630 can receive theplurality of selected region-of-interest (ROI) blocks 645. Theregion-of-interest (ROI) map generator module 630 can be configured togenerate a region-of-interest (ROI) map 650 including an indication forregion-of-interest (ROI) blocks that are of interest. Theregion-of-interest (ROI) map generator 630 can generate theregion-of-interest (ROI) map 650 as described above with reference toFIG. 3.

The video encoder 615 can receive the stream of video frames 635 and theregion-of-interest (ROI) map 650. The video encoder 615 can beconfigured to generate a compressed bit stream 655 based on theregion-of-interest (ROI) map 650. In one implementation, the videoencoder 615 can be configured to encode the memory blocks having acorresponding first region-of-interest (ROI) map bit value (e.g.,non-region-of-interest memory blocks) at a first bit rate (e.g., low bitrate) and the memory blocks having a corresponding secondregion-of-interest (ROI) map bit value (e.g., region-of-interest memoryblocks) at a second bit rate (e.g., high bit rate). In anotherimplementation, the video encoder 615 can be configured to encode thememory blocks having a corresponding first region-of-interest (ROI) mapbit value (e.g., non-region-of-interest memory blocks) at a firstquality (e.g., low quality) and the memory blocks having a correspondingsecond region-of-interest (ROI) map bit value (e.g., region-of-interestmemory blocks) at a second quality (e.g., high quality).

The artificial intelligence (AI) accelerator 605 and central processingunit (CPU) 610 operate substantially the same as described above withreference to FIG. 4 for non-object-based regions-of-interest (NOB-ROI)detection, and therefore will not be described further herein.

For non-object-based region-of-interest (NOB-ROI) detection, thecomputational workload of the video processing system 600 can be reducedbecause the object classification does not need to be performed forvariable rate video encoding. In addition, the reduced computationalworkload can result in a reduction in power consumption by the videoprocessing system 600. The non-object-based region-of-interest (NOB-ROI)can also advantageously be performed entirely in the artificialintelligence (AI) accelerator 605. In non-object-basedregion-of-interest (NOB-ROI), there is no need for sorting andnon-maximum suppression (NMS) and therefore the computational workloadof the central processing unit 610 can be reduced. The reduction of thecomputational workload of the central processing unit 610 can alsoreduce power consumption in the central processing unit 610. Inaddition, the bandwidth utilization of one or more communication linksbetween the artificial intelligence (AI) accelerator 605 and the centralprocessing unit 610 is advantageously reduced because data does not needto be transferred from the artificial intelligence (AI) accelerator 605to the central processing unit 610 for performing sorting andnon-maximum suppression (NMS). The reduction in data transmission fromthe artificial intelligence (AI) accelerator 605 to the centralprocessing unit 610 can also reduce power consumption.

Referring now to FIG. 7 an exemplary processing unit including a videoprocessing unit, in accordance with aspects of the present technology,is shown. The processing unit 705 can include one or more communicationinterfaces, such as peripheral component interface (PCIe4) 710 andinter-integrated circuit (I²C) interface 715, an on-chip circuit tester,such as a joint test action group (JTAG) engine 720, a direct memoryaccess engine 725, a command processor (CP) 730, and one or more cores735-750. The one or more cores 735-750 can be coupled in a directionring bus configuration. The one or more cores 735-750 can execute one ormore sets of computing device executable instructions to perform one ormore functions including, but not limited to, non-object-basedregion-of-interest (NOB-ROI) detection 220, threshold selection 225 andregion-of-interest (ROI) map generation as described above. The one ormore functions can be performed on individual core 735-750, can bedistributed across a plurality of cores 735-750, can be performed alongwith one or more other functions on one or more cores, and or the like.

The processor unit 705 can be a central processing unit (CPU), agraphics processing unit (GPU), a neural processing unit (NPU), a vectorprocessor, a memory processing unit, or the like, or combinationsthereof. In one implementation, one or more processors 705 can beimplemented in a computing devices such as, but not limited to, a cloudcomputing platform, an edge computing device, a server, a workstation, apersonal computer (PCs), or the like.

Referring now to FIG. 8, a block diagram of an exemplary processingcore, in accordance with aspects of the present technology, is shown.The processing core 800 can include a tensor engine (TE) 810, a poolingengine (PE) 815, a memory copy engine (ME) 820, a sequencer (SEQ) 825,an instructions buffer (IB) 830, a local memory (LM) 835, and a constantbuffer (CB) 840. The local memory 835 can be pre-installed with modelweights and can store in-use activations on-the-fly. The constant butler840 can store constant for batch normalization, quantization and thelike. The tensor engine 810 can be utilized to accelerate fusedconvolution and or matrix multiplication. The pooling engine 815 cansupport pooling, interpolation, region-of-interest and the likeoperations. The memory copy engine 820 can be configured for inter- andor intra-core data copy, matrix transposition and the like. The tensorengine 810, pooling engine 815 and memory copy engine 820 can run inparallel. The sequencer 825 can orchestrate the operation of the tensorengine 810, the pooling engine 815, the memory copy engine 820, thelocal memory 835, and the constant buffer 840 according to instructionsfrom the instruction buffer 830. The processing unit core 800 canprovide video coding efficient computation under the control ofoperation fused coarse grained instructions for functions such as regionof interest detection, bit rate control, variable bit rate videoencoding and or the like. A detailed description of the exemplaryprocessing unit core 800 is not necessary to an understanding of aspectsof the present technology, and therefore will not be described furtherherein.

The foregoing descriptions of specific embodiments of the presenttechnology have been presented for purposes of illustration anddescription. They are not intended to he exhaustive or to limit thepresent technology to the precise forms disclosed, and obviously manymodifications and variations are possible in light of the aboveteaching. The embodiments were chosen and described in order to bestexplain the principles of the present technology and its practicalapplication, to thereby enable others skilled in the art to best utilizethe present technology and various embodiments with variousmodifications as are suited to the particular use contemplated. It isintended that the scope of the invention be defined by the claimsappended hereto and their equivalents.

What is claimed is:
 1. A video processing unit comprising: anon-object-based region-of-interest detection neural network configuredto receive a video frame and generate a plurality of candidatenon-object-based region-of-interest blocks; a threshold selection moduleconfigured to receive the plurality of candidate non-object-basedregion-of-interest blocks and identify a plurality of selectednon-object-based region-of-interest blocks based on a predeterminedthreshold; and a region-of-interest map generator configured to receivethe plurality of selected non-object-based region-of-interest blocks andgenerate a region-of-interest map.
 2. The video processing unit of claim1, wherein the non-object-based region-of-interest detection neuralnetwork comprises a deep neural network including a regression networkconfigured to determine the plurality of candidate non-object-basedregion-of-interest blocks and corresponding probabilities of interest ofthe plurality of candidate non-object-based region-of-interest blocks.3. The video processing unit of claim 2, wherein the threshold selectionmodule is configured to identify the plurality of selectedregion-of-interest blocks having corresponding probabilities of interestgreater than the predetermined threshold.
 4. The video processing unitof claim 1, wherein the candidate non-object-based region-of-interestblocks comprise memory blocks of a predetermined matrix size of pixelvalues.
 5. The video processing unit of claim 1, further comprising: avideo encoder configured to receive the video frame and theregion-of-interest map and differential encode the video frame based onthe region-of-interest map.
 6. The video processing unit of claim 5,wherein the region-of-interest map comprises a plurality of bitscorresponding to a plurality of memory blocks of the video frame,wherein a first bit value indicates a corresponding memory blockcomprises a region-of-interest and a second bit value indicates acorresponding memory block comprises a non-region of interest.
 7. Thevideo processing unit of claim 6, wherein: memory blocks correspondingto bits of the region-of-interest map having the first bit value areencoded at a first bit rate; and memory blocks corresponding to bits ofthe region-of-interest map having the second bit value are encoded at asecond bit rate, wherein the second bit rate is less than the first bitrate.
 8. The video processing unit of claim 6, wherein: memory blockscorresponding to bits of the region-of-interest map having the first bitvalue are encoded at a first quality; and memory blocks corresponding tobits of the region-of-interest map having the second bit value areencoded at a second quality, wherein the second quality is less than thefirst quality.
 9. A video processing unit comprising: an artificialintelligence accelerator including; a non-object-basedregion-of-interest detection neural network configured to receive avideo frame and generate a plurality of candidate non-object-basedregion-of-interest blocks; a threshold selection module configured toreceive the plurality of candidate non-object-based region-of-interestblocks and identify a plurality of selected non-object-basedregion-of-interest blocks based on a predetermined threshold; aregion-of-interest map generator configured to receive the plurality ofselected non-object-based region-of-interest blocks and generate aregion-of-interest map; and an object-based region-of-interest detectionneural network configured to receive the video frame and generate aplurality of candidate object-based region-of-interest blocks; and acentral processing unit including; a sorting and non-maximum suppression(NMS) module configured to receive the plurality of candidateobject-based region-of-interest blocks, sort the plurality of candidateobject-based region-of-interest blocks for respective object types basedon associated probabilities, and combine multiple overlappingobject-based region-of-interest blocks to determine one or moreregion-of-interest bounding boxes for respective object types.
 10. Thevideo processing unit of claim 9, wherein the non-object-basedregion-of-interest detection neural network comprises a deep neuralnetwork including a regression network configured to determine theplurality of candidate non-object-based region-of-interest blocks andcorresponding probabilities of interest of the plurality of candidatenon-object-based region-of-interest blocks.
 11. The video processingunit of claim 10, wherein the threshold selection module is configuredto identify the plurality of selected region-of-interest blocks havingcorresponding probabilities of interest greater than the predeterminedthreshold.
 12. The video processing unit of claim 9, whereinobject-based region-of-interest detection neural network comprises adeep neural network including: a regression network configured todetermine the plurality of candidate object-based region-of-interestblocks and corresponding probabilities of interest of the plurality ofcandidate object-based region-of-interest blocks; and a classificationnetwork configured to determine a corresponding object type of theplurality of candidate object-based region-of-interest blocks.
 13. Thevideo processing unit of claim 9, wherein the artificial intelligenceaccelerator is configured to selective generate the plurality ofcandidate non-object-based region-of-interest blocks by thenon-object-based region-of-interest detection neural network or generatethe plurality of candidate object-based region-of-interest blocks by theobject-based region-of-interest detection neural network.
 14. The videoprocessing unit of claim 13, further comprising: a video encoderconfigured to receive the video frame and receive the region-of-interestmap or one or more region-of-interest bounding boxes, and differentiallyencode the video frame based on the region-of-interest map or the one ormore region-of-interest bounding boxes.
 15. A method of video processingcomprising: generating a plurality of candidate non-object-basedregion-of-interest blocks for a video frame, wherein each of thecandidate non-object-based region-of-interest blocks include acorresponding confidence score; identifying a plurality of selectedregion-of-interest blocks comprising candidate non-object-basedregion-of-interest blocks each of which have a confidence score greaterthan a predetermined threshold score; and generating aregion-of-interest map based on the plurality of selectedregion-of-interest blocks.
 16. The method of video processing accordingto claim 15, wherein generating the region-of-interest map comprises:determining, for memory blocks of video frame, if a given memory blockis a selected region-of-interest block; setting a bit value in theregion-of-interest map corresponding to the given memory block to afirst value if the given memory block is one of the plurality ofselected non-object-bases region-of-interest block; and setting the bitvalue in the region-of-interest map corresponding to the given memoryblock to a second value if the given memory block is not one of theplurality of selected region-of-interest blocks.
 17. The method of videoprocessing according to claim 16, further comprising: differentiallyencoding the video frame as a compressed bit stream based on theregion-of-interest map.
 18. The method of video processing according toclaim 17, wherein differentially encoding the video frame comprises:encoding memory blocks corresponding to bits of the region-of-interestmap set to the first bit value at a first bit rate; and encoding memoryblocks corresponding to bits of the region-of-interest map set to thesecond bit value at a second bit rate.
 19. The method of videoprocessing according to claim 15 further comprising: selectivelygenerating the plurality of candidate non-object-basedregion-of-interest blocks or generating a plurality of candidateobject-based region-of-interest blocks, wherein the candidateobject-based region-of-interest blocks include a correspondingconfidence score and a corresponding object type; sorting the pluralityof candidate object-based region-of-interest blocks of correspondingobject type based on the corresponding confidence score; and combiningmultiple overlapping one of the plurality of candidate object-basedregion-of-interest blocks to determine one or more region-of-interestbounding boxes.
 20. The method of video processing according to claim19, further comprising: encoding memory blocks inside the one or moreregion-of-interest bounding boxes at a first bit rate; and encodingmemory blocks outside the one or more region-of-interest bounding boxesat a second bit rate.